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Hyperspectral Imaging Methods And Devices 



Provisional Patent application 

Inventors: Andreas C. Coppi, Groton, CT; Richard A. DeVerse, Kailua Kona, HI; 
William G. Fateley, Manhattan, KS; Frank B. Geshwind, Madison, CT. 

The present invention relates to methods and devices for hyperspectral imaging. 
One embodiment, called "NSTIS", for Near-infrared Spectral Target Identification 
System, is a DMD-modulated-aperture imaging spectrograph system, operable in 
the near-infrared. It includes fore optics for imaging a scene of interest, a micro- 
mirror array spatial light modulator (SLM or DMD), a diffraction grating and an 
infrared camera, together with the necessary optics to transfer the image 
between these components, and electronics to drive the SLM. 

The present invention related to further embodiments, details and improvements 
on such a system that has been described in US. Patent Application Serial No. 
09/798,860, filed March 1, 2001, titled "System and Method for Encoded Spatio- 
Spectral Information Processing,", U.S. Patent No. 6,392,748, US Patent No. 
6,046,808 and US Patent No. 6,128,078. These patents and application are 
herby incorporated by reference. 

One Possible General layout is illustrated in figure 1 

For further illustration, figures 6 and 7 show photographs of a Plain Sight 
Systems DMD modulated aperture imaging spectrograph system built to operate 
in the visible wavelength range. 

In a preferred embodiment, a scene or object of interest is imaged onto a spatial 
light modulator. The spatial light modulator is used to pass or reject the spatial 
resolution elements of the image. Spatial resolution elements that are so 
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selected propagate through an imaging spectrograph system and are spectrally 
imaged onto a focal plane array sensor. 

The digital micromirror device (DMD) spatial light modulator (SLM) presents a 
challenge in that typical imaging optics have image planes that are perpendicular 
to the optical propagation axis Z. When the object is imaged by these 
conventional optical imaging systems onto the DMD then the image now 
becomes an object for the imaging spectrograph and the DMD can selectively 
relay spatial resolution elements of this object-image onto the imaging 
spectrograph portion of the system. This however requires that special optical 
systems capable of handling this condition of the tilted object. Furthermore the 
resultant image is tilted. Generally speaking, this results in poor optical 
performance for imaging systems and typical imaging systems cannot cope with 
this condition at low F numbers of operation. Figure 2 shows an example of the 
tilted object plane imaged onto a resultant tilted image plane using a simple 
image relay lens system. 

Figure 4 shows a model of a lens imaging a source onto the DMD SLM through a 
Near Infrared optimized total internal reflection (TIR) prism assembly that 
functions to reduce effective F# of operation by creating a greater path 
divergence of the incoming and outgoing beam of light. This prism assembly can 
be so constructed that it functions to separate both the "on" deflected light (+10 
degree mirror position) and the "off light (-10 degree mirror positions). Figure 3 
only shows the "on" or (+10) degree position of the mirrors so as to direct the 
light to the subsequent optical system such as the imaging spectrograph. 

One embodiment of a DMD enabled staring spectral imaging system is shown in 
Figure 5. This configuration uses a typical imaging lens to present an image of 
the object under observation to the DMD SLM through a near-infrared (NIR) TIR 
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prism assembly. The spatial resolution elements that are selected to pass onto 
the relay lens system and transmission diffraction grating and are presented to 
the camera 2D sensor array or focal plane array(FPA) 

One can also employ a system where two lens sets are used in conjugate 
positions such that the first lens set collimates the light from the DMD SLM to the 
diffraction optical element and the second lens focuses the dispersed light onto 
the FPA. 

Note that the lenses are either refractive elements or reflective optical systems. 

A preferred embodiment uses all reflective optical components so that the 
system is not limited by the spectral range of transmission of the refractive 
elements used in the system. All reflective optical elements also permit chromatic 
error free operation. An improved embodiment of the staring hyperspectral 
imaging system uses a special optical relay system developed by plain sight 
systems to place the image onto the DMD in the preferred manner where the 
image plane that is perpendicular to the optical propagation axis Z is made tilted 
for presentation to the DMD without a TIR prism assembly. Figure 8 below shows 
an optimized all reflective model of an optical systems that takes an object plane 
and presents a tilted image at precisely the correct angle to the DMD so as to 
allow the use of typical optical sub-systems to receive the spatial resolution 
elements that are selected to propagate through the system in a conventional 
manner. This is important because most spectral imaging systems are very 
sensitive to the degree of collimation for resolution elements impinging upon a 
diffraction element. If the DMD object that is a result of the primary object image 
is tilted the spectral imaging system has difficulty if spectrally re-imaging the 
primary object at the FPA. 

In another embodiment, the reflective optics described are used with a TIR prism. 
Figure 9 shows an extension of the system shown in Figure 8 for use with a TIR 
prism assembly. 
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Figure 10 shows one preferred embodiment of the staring hyperspectral imaging 
system as designed in one PSS prototype application. A refractive camera lens 
receives the light from the source object and images it onto the object plane of 
the all reflective transfer optical system that takes this conventional 
perpendicularly presented image with respect to the optical axis from the 
conventional camera lens system and presents it through the TIR prism to the 
DMD micro-mirror plane in a tilted fashion with respect to the optical axis such 
that the selected spatial resolution elements can propagate to the imaging 
spectrograph portion of the system. This condition allows more typical imaging 
optical systems to be used and improves the collimation of all of the desired 
spatial resolution elements used in the DMD SLM. Improving the collimation 
improves the effectiveness of the diffractive element and improves the 
subsequent image quality as it is focused by the focusing optical elements in the 
spectrograph portion of the system. 

Figure 11 shows a solid model of the system shown in Figure 10 

Figure 12 shows the imaging spectrograph portion of the PSS prototype 
hyperspectral imaging system using conventional refractive optical elements as 
made possible by the PSS proprietary tilted optical relay design to the DMD SLM. 

Figure 13 shows a map of DMD SLM micromirrors to the FPA. The DMD micro- 
mirror representation is shown on the left of the figure 13 in black. The 
corresponding spectrally dispersed images of the of the micro-mirror column are 
shown in black at the representative FPA on the immediate right. As the column 
of micro-mirrors that are selected to be in the "on" condition move from left to 
right so to does the spectrally dispersed images of the column move from left to 
right on the FPA for a given spectral range of operation. One can also select a 
multi-tude of columns simultaneously that have the effect of presenting a mulit- 



Page4of 13 



tude of overlapping spectral images onto the FPA. This method is called multi- 
plexing and is described elsewhere. 

Optics for Hadamard Encoded Apertures 

Hadamard optical systems utilize spatially encoded apertures that can be 
employed either at the entrance aperture of optical system, the exit aperture or 
both. They have the common attribute that they encode the available aperture 
spatially where the spatial resolution elements that make up the encodement 
dictate the spectral, spatio-spectral or spatial resolution elements that propagate 
through the optical system including diffractive optical elements and on to the 
sensor or exit aperture. These masks have some spatial extent that places 
special requirements on the optics of the system. As the encodement mask 
grows either by longer length encodements with fixed sub-apertures or as the 
sub-aperture dimension grows for a fixed encodement length, the spatial 
resolution elements making up the sub-apertures in the encodement mask depart 
from the optical axis. When the resolution elements depart from the optical axis 
or paraxial condition it becomes important to employ optics that can image the off 
axis resolution elements without inducing excessive aberrations that degrade the 
performance or cripple the advantages gained by HT multiplexing. 

Typically the optical path for conventional monochromators begins with a 
source that is focused onto an aperture plane that has a large aspect ratio 
aperture know as a slit. This slit is often very small in extent in the dispersion 
plane compared to the other extent in the spatial plane. However, it is not 
required that this aspect ratio is large. If the aspect ratio is close to 1 then simple 
spherical optics can be employed that perform well as long as the departure from 
the optical axis is kept to a minimum. However, most monochromators have a 
large aspect ratio in order to increase the opportunity to maximize throughput, 
and detectors must be able to "see" the large extent of the slit aperture. The light 
entering the slit aperture is then dispersed and focused onto an exit slit aperture. 
Monochromators are only required to perform well on the optical axis and do not 
typically employ optics that can manage rays that depart from the optical axis in 
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the plane of dispersion as required by HT multiplexing instruments. To employ 
encoding techniques the optical system is required to utilize optical performance 
attributes normally found only in imaging and spectral imaging systems. This 
requirement is driven by the extent of the encoding mask. The extent of the 
encoding mask is governed by the diffraction limit of the wavelengths within the 
bandpass, the encodement length N and the attributes of the optical system. 

In a conventional dispersive spectrometer the radiation from a source is 
collected and separated into it's individual spectral resolution elements by a 
spectral separator such as a diffraction grating or prism and then is collected and 
focused for spatial presentation on a focal plane. The dispersive spectrometer 
uses a single exit slit to select one spectral resolution element of N spectral 
resolution elements for measurement by the detector. The Hadamard transform 
spectrometer (HTS) uses an array of slits (i.e. a mask) at the focal plane to select 
one more than half, (N+1)/2, of the spectral resolution elements at the focal plane 
for measurement by the detection system. The optical challenge to effect an HT 
multiplexing spectrometer is to collect all of the spatially distributed individual 
band pass images of the entrance slit and transfer them to as small detector as 
possible. One desires to keep the area of the detector at a minimum as the noise 
of many detectors increases with the square of the area. If the optics are able to 
illuminate a single detector element with all of the available light impinging upon 
the focal plane containing the spatially distributed images of the slit for each of 
the N band pass resolution elements, a multitude of spectral resolution elements 
can be measured simultaneously using a single detector element. This 
arrangement results in a multiplexing spectrometer. The recovery of N spectral 
resolution elements requires measuring the detector response for N different 
encodements of (N+1)/2 open mask elements. The raw data is recorded as the 
detector response versus encodement number and is called an encodegram. 
Hadamard transformation of the encodegram yields the spectrum. 
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The History of Applied Hadamard Multiplexing 

The Hadamard transform instruments developed in the 1960s and 1970s 
employed moving masks. Significant problems such as misalignment and 
jamming associated with a moving mask led to a reputation of poor reliability and 
contributed to a dormant period in the development of HTS and HTI. Interest was 
rekindled in the 1980s using stationary Hadamard encoding mask based on 
liquid crystal (LC) technology. The first generation 1 D stationary Hadamard 
encoding mask was a cholesteric LC with N = 127 mask elements and used 
polarization as its operating phenomenon. Two parallel polarizers and rotation or 
lack of rotation of the polarized radiation generated the opaque and transparent 
states, respectively. The second generation 1D stationary Hadamard encoding 
mask was fabricated using a polymer dispersed liquid crystal (PDLC) material 
with N = 255 mask elements and used light scattering as its operating 
phenomenon. The PDLC contained LC droplets dispersed in a polymer matrix 
whose index of refraction matched the index of refraction in one direction in the 
birefringent LC droplet. Alignment of the LC droplets optical axis under an 
applied voltage removed discontinuities in index of refraction at the polymer 
matrix/LC interface to generate a good transparent state while random 
orientation of LC droplets in the polymer matrix generated the opaque state from 
light scattering by the discontinuities in index of refraction at the polymer 
matrix/LC droplet interface. A 2D stationary Hadamard encoding mask was also 
based on LC technology. A fero-electric liquid crystal (FLC) positioned between a 
pair of polarizers with perpendicular orientation operated as an electro-optic half- 
wave plate when a + value of applied voltage rotated the plane of polarization by 
90 degrees to produce the transparent state and a - value of applied voltage left 
the plane of polarization unaltered to produce the opaque state. 

Development based on stationary Hadamard encoding masks continued 
in the 1990s and a 2D moving Hadamard encoding mask was also fabricated 
and used to perform imaging in the near-infrared and mid-infrared spectral 
regions. Note that the mid-infrared spectral region is not generally accessible via 
Hadamard encoding masks based on LC technology since any LC material is 
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expected to have strong absorption bands in the mid-infrared spectral region. 
From the mid 1990s to the present the stationary Hadamard encoding mask of 
choice for the visible and near-infrared spectral regions has been the digital 
micro-mirror device (DMD), a device based on micro-optoelectromechanical 
systems (MOEMS) technology and developed by Texas Instruments for projector 
display applications. One DMD format incorporates 508,800 micro-mirrors in a 
848 column by 600 row array that is 14.4mm wide by 10.2mm high. Each 
individual micro-mirror is 16microns square and adjacent micro-mirrors are 
separated by a 1 micron gap. (need Greekl Ic letter mu followed by m here) The 
micro-mirrors are individually addressable and rotate by +10 or -10 degrees 
about the diagonal axis to produce binary "on" and "off' states. The on state has 
T t determined by the mirror reflectivity and approaches 1 while the off state 
approaches T 0 = 0. The ideal condition of on and off is not realized due to 
diffraction of the light off of the small and periodic features of the micro-mirror 
device. The DMD is an array of spatial resolution elements that may be selected 
as groups of super-resolution elements or as individual resolution elements 
consisting of a single micro-mirror. The DMD resolution elements are realized as 
spectral resolution elements in the spectrometer with the columns attributed to 
the frequency or wavelength dimension and the rows attributed to the slit height 
dimension. The DMD resolution elements are utilized as spatio-spectral 
resolution elements in the imaging spectrograph with the columns as the 
frequency or wavelength dimension and the rows as a vertical spatial dimension 
with the horizontal spatial dimension being accessed, if desired, by translating 
the sample relative to the imaging spectrograph. The DMD resolution elements 
are spatial resolution elements in the imager with the columns for the horizontal 
dimension and the rows for the vertical dimension and the frequency or 
wavelength dimension provided by other instrumentation. If a photo-acoustic 
detection system is present then the depth dimension of the sample may also be 
accessed by changing the modulation frequency used in the photo-acoustic 
detection system. 

Some important features of HTS to keep in mind are: 
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(1) It is a dispersive technique using a spectral separator 

(2) It is a multiplexing technique using a single-element detector 

(3) It uses a Hadamard encoding mask (multi-slit array) in the focal 
plane 

(4) It sends one more than half the resolution elements to the single- 
element 

detector in an encodement 

(5) It uses a number of encodements equal to the number of resolution 
elements desired and the number of mask elements (pixels) in the 

stationary Hadamard encoding mask (a moving Hadamard 
encoding mask 

has 2N - 1 mask elements) 

(6) It has each encodement contain a different combination of one more 
than half the resolution elements. 

(7) It has as its primary data the encodegram, a record of detector 
response versus encodement number. 

(8) It uses a FHT of the encodegram to decode the encodegram and 
generate the spectrum or image. 

Note that all of these except for item (1) also apply to HTSI and HTI. The DMD 
promises to be the best Hadamard encoding mask yet developed for the visible 
and near-infrared spectral regions. However, its potential applications in 
spectrometry and imaging are by no means limited to HT techniques since the 
information corresponding to any micro-mirror in the DMD may be included in or 
excluded from any measurement as desired by the investigator . An instrument 
with no moving parts other than the micro-mirrors in the DMD promises to 
provide a compact and robust instrument for operation in potentially hostile 
environments ranging from process control to outer space. It is our belief that the 
combination of a DMD with a single-element detector may provide an important 
advance in spectroscopic instrumentation and that instrumentation based on the 
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DMD may lead to a host of environmental, industrial, medical, and military 
applications. 

Plain Sight Systems Technology Overview 

• Multiplexed Spectral and Hyperspectral data and imagery 

• PSS's MEMS-based multiplexing 

• NstisApp software for interfacing to NSTIS 

• Optical-digital domain Endmember processing 

• Advanced ATR and Feature extraction algorithms 
Plain Sight Systems Technology, Some Advantages 

• Extended of hours of operability 

• Operability where conventional single slit spectral imaging systems 
fail 

• Improved signal to noise ratios 

• Increased sensitivity for chemistry and features of interest in 
hyperspectral imagery and spectral data 

• Better Pd/Pe ratios 

• Faster information delivery 

• Smaller bandwidth for telemetry of data 

• Reduction in post processing of data 



Spec 

• Sampling spectral 

• Spectral resolution 

• Working F# 

• Data collection time 

• Datacube 



Performance 
1.5625nm 
< 10nm 
<5.6 

10 sec / datacube 
(typical. Depends 
on light levels) 
up to 512x512x532 
(256x256x266 also) 
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Vis-NIR: What it Does 

Silicon spectral range 400nm-1100nm 

Cube: 400X600spatial X 255 spectral 



• Staring Hyperspectral imaging system 

• Programmable information collection 

• Improves SNR 

• Faster image cube acquisition 

• Night vision capable 

• Variable resolution 

• Adaptive Chemometrics in the optical domain 

Vis-NIR: Applications 

• Spectral imagery 

• Dermatology 

• Pathology 

• Chemical detection in scenes 

• Target recognition and identification 

• Process control - Agriculture/Industry 

NSTIS: What it Does 

Spectral range InGaAs: 900nm-1700nm 

Hyperspectral cube as large as: 512X51 2spatial X 266 spectral 

• Staring Hyperspectral imaging system 

• Programmable information collection 

• Improved SNR over raster scanning methods 

• Faster image cube acquisition 

• Night vision capable 

• Variable resolution 

• Adaptive Chemometrics in the optical domain 
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NSTIS: Applications 

• NIR Spectral Imagery 
Pharmaceutical Classification 

• Process control Agriculture/Industry 

• ' Plastics sorting 

• Target identification and classification 

• Adaptive chemometry in images 

• Night-time spectral imagery 

• Taking funny pictures of your friends 

• Development of chemometric methods 
HSE: What it Does 

• Allows the user to quickly view and manipulate hyperspectral data cubes in a 
flexible interface. 

• Develop chemometry and applied methodology for complex sample matrices and 
environments 

• Application of LDB analysis 
HSE: Uses 

• Principal Component Identification 

• LDB 

• Hierarchical Clustering 

• Normalization 

• Intuitive Training Set Interface 
NSTIS App Software 

• For use with NSTIS to perform on the fly scene selection and averaging 

• Allows easy access to a hyperspectral datacube 

• Windows based GUI application for hyperspectral image acquisition. 

• Optimized hyperspectral data acquisition 

• Preview of spectra from image regions, including single beams, ratios, dark 
noisecorrections and 100% line images. 

• fast local-orthogonal component analysis algorithm for quick data exploration. 
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In a preferred embodiment the system includes software for acquisition of hyperspectral 
data, and a user interface for interactively selecting regions from a 2D projection of the 
3D hyperspectral datacube. Such an embodiment can also include software for 
computing features of the selected regions, and display of the 3D datacube, project into 
2D via the selected features. One example of a way to do this would be to compute the 
mean spectral vector in each region, and then compute a Gram Schmidt orthogonalization 
of the selected vectors. When there are 3 selected vectors, the output of the Gram 
Schmidt algorithm can be sued to compute 3 spectral inner products over the datacube, 
and the results used to render an RGB image of the datacube. The process can then be 
iterated, providing the user the ability to select regions in the original and the processed 
2D projections. 
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Figure 1 : A general layout for NSTIS 



Figure 3. Example of asymmetric imaging relay spectrograph using simulated DMD 
tilted object, two paraxial lenses, a grating and a tilted FPA at the image plane. 




F2.72 OPERATION PSS NIR TIR 10 DEGREE OMO 



LENS HflS NQ TITLE. 
WED JUL 1 6 2803 



3D LRYOUT 



CONPICURQTIQN 1 OF 1 



Figure 4. Imaging lens and TIR prism example for improving F# of operation using the 
DMD as an SLM to select spatial resolution elements of an image to pass to subsequent 
optical elements of the imaging or spectral imaging system. 



Figure 5. A typical camera lens (near the upper left of the drawing, shown in green in 
color renderings) receives light from the source object and re-images the object onto the 
DMD where the plane of micro-mirrors are surface normal to the optical propagation axis 
Z. The DMD can then be used to select which spatial resolution elements of the image 
will propagate through the TIR prism and into the relay lens, through the transmission 
grating and spectrally imaged onto the camera sensor or focal plane array (FPA), shown 
on the right of the drawing). 



Figure 6: Visible System DMD module and imaging objective 




Figure 7: Visible System complete prototype 



TILT OBJ 
OPTICAL SYSTEM LAYOUT 



UNITS: MM 
DES: CDH 



50.8 




Figure 8. Optimized model of an all reflective image delivery system for a DMD enabled 
SLM spectral imaging system or imaging system. The object to be imaged onto the DMD 
is shown at the bottom of the image perpendicular to the optical propagation axis. The 
design and placement of the relay optical system presents a tilted image appropriate for 
presentation to the DMD SLM such that the subsequent spatial resolution elements so 
selected propagate perpendicular the the DMD micro-mirror plane. This allows 
conventional optical systems to be employed subsequently to the DMD such that image 
quality is maintained to a high degree of coherence and performance. 



Figure 9. An optical relay system optimized for use with a TIR prism assembly and the 
DMD SLM such that the tilted image presented to the DMD reflects selected resolution 
elements perpendicular to the micro-mirror array plane. This allows conventional optical 
systems such as refractive collimating lenses to be used. 




Figure 1 1. A solid model of the PSS prototype staring hyperspectral imaging system. 




Figure 12. imaging spectrograph in the PSS prototype NSTIS using refractive COTS lens 
system made for conventional NIR imaging applications. 
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Figure 13, Example of the mapping of micro-mirror columns to the dispersed spectral 
images at the focal plane of a 2D array sensor, FPA or camera. 
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